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Overview 


This rubric is a tool for evaluating reading/language arts 
instructional materials for grades K-5. Based on rigorous 
research, the rubric can be used by state-, district-, and school- 
level practitioners and by university faculty who review instructional 
materials. The rubric is organized by content area for grades K-2 
and for grades 3-5. Each content area (for example, writing) 
includes a list of criteria for evidence-based practice that the 
instructional materials are expected to include. Each criterion is 
aligned to recommendations from six What Works Clearinghouse 
practice guides, and a 1-5 scale is used to rate how well the 
criteria were met. Guidance for when and how to use the rubric — 
including facilitator responsibilities, professional learning for 
reviewers, and ways to use the ratings — is also provided. 
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Introduction 


The implementation of effective instructional materials, such as a core reading program, 
by a qualified teacher is an important part of improving students’ reading achievement. 
But selecting those instructional materials can be time-consuming. Regional Education- 
al Laboratory (REL) Southeast created this rubric for evaluating reading/language arts 
instructional materials for kindergarten to grade 5 to help educators assess how consistent 
instructional materials (including core reading programs and reading intervention pro- 
grams) are with the scientific research on reading instruction. This document suggests pro- 
cedures for using the rubric to review instructional materials and inform decisions at the 
state, district, or school level about reading/language arts instructional materials for grades 
K-5. Members of the Improving Literacy Research Alliance from the Florida Department 
of Education requested the rubric to assist in their instructional materials review process. 
Educators in other states may also find the rubric to be beneficial. 

Need for rigorous review of instructional materials 

Teaching is an interaction of a teacher’s instructional strategies, classroom instructional mate- 
rials, and students. Recently, the importance of instructional materials has been overlooked 
(Chingos & Whitehurst, 2012), and the pedagogical connection between lesson objectives and 
instructional materials is rarely coherent (Foorman, Francis, Davidson, Harm, & Griffin, 2004). 

Most publishers have revised reading/language arts curriculum materials to support imple- 
mentation of contemporary state standards. Materials in grades K-2 focus on support for 
teaching foundational reading skills in order to develop competent readers and build students’ 
capacity to comprehend a range of text types across many disciplines. The focus for grades 
3-12 materials shifts from developing reading skills to applying those skills to understand 
complex texts. In the upper elementary grades, instructional materials include equal expo- 
sure to literary and expository text, an emphasis on developing academic vocabulary, and 
attention to fostering students’ ability to analyze and comprehend increasingly complex text. 

The U.S. Department of Education’s What Works Clearinghouse provides rigorous reviews 
of research studies of reading interventions (for example, James-Burdumy et al., 2012). 
However, there are few empirical evaluations of core reading programs — materials to be 
used with all students (Chingos & Whitehurst, 2012; see the section on research support 
for the development of the rubric for more information about the What Works Clearing- 
house). Educators thus need to be able to determine which reading/language arts instruc- 
tional materials are aligned with research. This rubric helps address this need. 

When to use the rubric 

The rubric can be used in many ways. First, a state or district can use it to develop a list 
of approved materials for purchase by districts and schools that are adopting new materi- 
als. Second, it can be used when a district or school plans to purchase new instruction- 
al materials to determine whether the new reading materials are aligned with research. 
Third, a district or school can use the rubric to review currently implemented instructional 
materials to determine how well they align with reading research. For that purpose, gaps 
in current instructional materials are identified so that materials may be modified or com- 
bined or plans may be made to purchase new or additional materials. Finally, the rubric 
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can be used when instructional materials are developed (for example, at the district level) 
to ensure that they align with reading research. 

Under any of these circumstances, a state, district, or school can modify the rubric. For 
example, if a state or district has additional or different standards, they can be added to or 
modified in the rubric. The rubric is based on empirically derived research and can be used 
with any contemporary state standards for grades K-5. 

Knowledge and experience required to use the rubric 

A team of reviewers use a rating scale on a rubric to conduct reviews of the instructional 
materials. Because the value of the ratings depends on the judgments of the review team 
that uses it, members of the review team need to have knowledge of instructional design, 
reading/language arts content, and pedagogy. Educators at the state, district, and school 
levels who have expertise in grades K-5 reading/language arts content and experience as 
classroom teachers — and thus are users of instructional materials — can use the rubric, as 
can university faculty with expertise in instructional design, content, and pedagogy who 
are working in partnership with school districts. 

For elements of instructional design, understanding how a curriculum is created is import- 
ant. That is, it is imperative that the review team understand how to identify a system- 
atic scope and sequence, how goals and objectives are related, what the elements of an 
organized lesson are, and how to align materials and embed formative assessments. The 
content is what is taught during reading/language arts instruction (such as phonics, spell- 
ing, comprehension, and writing). Pedagogy is how the content is taught (such as explicitly 
using routines or differentiated instruction). Differentiated instruction materials include 
activities that address both intervention for students with special learning needs and 


Figure 1. Elements of design for reading/language arts instructional materials 


Instructional design 
Scope and sequence 
Goals and objectives 
Lesson organization 
Alignment and assessment 


Content-what is taught 
Print concepts 
Phonological awareness 
Phonemic awareness 
Phonics 

Advanced word analysis 
Fluency 

Language (vocabulary and 
academic language) 

Comprehension 

Spelling 

Writing 

Speaking and listening 


\ 


Pedagogy-how it is taught 
Explicit 
Systematic 

Coordinated instructional 
sequences and routines 

Scaffolded 
Feedback 
Differentiated 


Source: Authors' construction. 
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extension/enrichment for students ready for further work. Salient features of instructional 
design, reading/language arts content, and pedagogy are shown in figure 1. 

Facilitating use of the rubric 


There are several considerations to facilitate use of the rubric, including selecting a facili- 
tator and review team, providing professional learning for the reviewers, and developing a 
checklist of materials to be reviewed. 


Selecting a facilitator and review team 


The instructional materials review process works best when the administrator in charge of 
the process selects a dedicated facilitator to lead the effort and review the rubric in detail 
before the review process begins. The facilitator may be an educator at the state, district, or 
school level who is knowledgeable in instructional design, content, and pedagogical research 
as well as in instructional materials review policies, procedures, and implementation. The 
facilitator should also collect relevant sample materials and possible published sources of 
evidence before convening a meeting with the review team. The facilitator should be a 
careful listener who can lead and structure discussions around decisionmaking processes for 
members of the review team. Examples of facilitators and their purpose for using the rubric 
include a curriculum director at a state department of education coordinating adoption of a 
new curriculum, a district’s chief academic officer organizing a team to review instructional 
materials under consideration for purchase, and a school leader facilitating the review of 
currently implemented instructional materials as part of a needs assessment. 
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The facilitator has several responsibilities during the review process: 

• Recruiting and convening the review team (3-10 educators, depending on the 
quantity of materials to be reviewed). 

• Ensuring that reviewers have enough time for reviews. 

• Developing and implementing professional learning for the reviewers. 

• Determining initial inter-rater reliability. 

• Assigning instructional materials to reviewers. 

• Developing and providing checklists of requirements for publishers. 

• Considering pros and cons of reviewing hard copy versus digital instructional 
materials. 

• Facilitating consensus meetings. 


The review team can be configured in various ways, depending on the purpose. For 
example, states could call on an expert review team from a research university for a 
state-level review, districts could draw on their reading/language arts instructional experts 
for a district-level review, and schools could enlist reading coaches, veteran teachers, 
and district-level support to use the rubric. Each review team needs to have knowledge 
of instructional design, reading/language arts content, and pedagogy, and be free of bias. 
For example, facilitators would want to exclude from the review team anyone with known 
financial conflicts. Facilitators might consider using a conflict-of-interest disclosure form to 
reveal perceptions of bias, which could be discussed and mitigated. 


It can take more than 40 hours to review one set of instructional materials, so the facilitator 
should ensure that the review team has the time necessary to thoroughly review the assigned 
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instructional materials. It is beneficial for individual reviewers to be assigned to review instruc- 
tional materials for which they have the most knowledge and experience (that is, K-2 or 3-5). 

Providing professional learning for the reviewers 

In addition to recruiting reading/language arts educators who have the knowledge, expert 
ence, and objectivity needed to conduct the reviews, the facilitator provides standardized 
professional learning to orient the reviewers to the rubric. Face-to-face professional learn- 
ing is ideal; however, a virtual meeting can also be successful. 

The content of the professional learning developed by the facilitator includes: 

• An overview of the rubric. 

• How to conduct a review. 

• Whom to contact with questions. 

• Guidance on the number of hours it will take to review instructional materials 
and to develop a schedule of interim and final deadlines to complete the reviews 
(this will depend on the quantity of materials being reviewed). 

• A practice opportunity to review a sample set of instructional materials. 

The practice opportunity for reviewers to use the rubric before the actual review process 
begins serves two purposes. First, it provides a chance for reviewers to discuss the experience, 
obtain answers to questions, and feel confident using the rubric. Second, it is an opportu- 
nity to establish the initial reliability and calibration of reviewers to increase consistency. 

Before the review begins, the facilitator should assign each set of instructional materials 
to two different reviewers to determine inter-rater reliability. There are two instances in 
which the facilitator will need to help determine consensus after individual reviews are 
completed. First, if inter-rater reliability is low, a third reviewer will need to review the set 
of instructional materials with low inter-rater reliability in order to resolve discrepancies 
(see the appendix for how to calculate inter-rater reliability). Second, if a reviewer cannot 
decide what rating to give an item, the facilitator will convene a consensus meeting and 
guide a conversation among reviewers using appropriate small group discussion procedures, 
such as active listening and shared discussion. 

Developing a checklist of materials to be reviewed 

The facilitator may develop a checklist of required items that the publisher of the instruc- 
tional materials under consideration needs to submit in order for a comprehensive review 
to occur. This checklist is provided to both the publisher and the review team. Items for 
the checklist may include: 

• An overview (for example, a presentation) of the program, sample materials, and 
video of demonstration lessons. 

• A detailed scope and sequence, including an alignment to adopted state standards 
and to research. 

• A mapping of the location of content in the materials to criteria on the rubric. 

• A list of the titles of books and documentation of their range and complexity. 

Instructional materials can be reviewed in hard copy or electronically; there are benefits 
and challenges to each approach. For example, with digital materials, it may be difficult to 
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look across grades to see the continuum of instruction for particular skills when going back 
and forth between different electronic documents. But publishers can also prepare digital 
materials that are easy to manipulate and navigate. 

If materials are to be reviewed electronically, the facilitator should verify that the review 
team has access to the necessary hardware and software. For example, computers must 
support downloads of large files and the simultaneous running of multiple programs and 
display of multiple browser windows. 

Research support for the development of the rubric 


The rubric is based on what rigorous research indicates is the most effective way to teach 
reading/language arts. The What Works Clearinghouse identifies research studies that 
provide credible evidence of the effectiveness of a given practice, program, or policy (coh 
lectively referred to as “interventions”) and disseminates summary information and reports 
on its website (http://ies.ed.gov/ncee/wwc). The content of this rubric is based on six 
What Works Clearinghouse practice guides that pertain to content areas within reading/ 
language arts: 

• Assisting Students Struggling with Reading: Response to Intervention (Rtl) and 
Multi-Tier Intervention in the Primary Grades (Gersten et al, 2008). Abbreviated 
as RTI. 

• Foundational Skills to Support Reading for Understanding in Kindergarten through 3rd 
Grade (Foorman et al., 2016). Abbreviated as FR. 

• Improving Adolescent Literacy: Effective Classroom and Intervention Practices: A 
Practice Guide (Kamil et al., 2008). Abbreviated as Adol. 

• Improving Reading Comprehension in Kindergarten through 3rd Grade: A Practice 
Guide (Shanahan et al., 2010). Abbreviated as RC. 

• Teaching Academic Content and Literacy to English Learners in Elementary and 
Middle School (Baker et al., 2014). Abbreviated as EL. 

• Teaching Elementary School Students to Be Effective Writers: A Practice Guide 
(Graham et al, 2012). Abbreviated as Writing. 
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Abbreviations used in the rubric that are related to recommendations and action steps: 

• Rec. refers to the recommendation in the practice guide. 

• # refers to the specific action step for a recommendation. If no # is indicated, then 
all action steps for a recommendation apply. 

Abbreviations used in the rubric that are related to spelling patterns: 

• CV is consonant vowel. 

• CVC is consonant vowel consonant. 

• CVCC is consonant vowel consonant consonant. 

• CCVC is consonant consonant vowel consonant. 

• CVe is consonant vowel final e. 


Using the rubric 


The rubric has separate sections for grades K-2 and grades 3-5, each with a subsection on 
content area and overall rating. 
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Content areas 


The content areas for grades K— 2 are foundational reading skills (print concepts, phono- 
logical awareness, phonemic awareness, phonics, and fluency); reading comprehension for 
literary and informational texts; writing development and skills; speaking and listening 
development and skills; and language development and skills (academic language skills and 
vocabulary). The content areas for grades 3-5 are foundational reading skills (advanced 
word analysis such as affixes, Greek and Latin roots, and syllable patterns); reading com- 
prehension for literary and informational texts and text complexity; writing development 
and skills; speaking and listening development and skills; and language development and 
skills. 

Each content area includes a list of criteria for evidence-based instructional practice. Each 
reviewer records findings based on the extent to which the criteria were met using a 1-5 
rating scale: 

• 1 indicates the criterion was not met. 

• 2 indicates the criterion was partially met. 

• 3 indicates the criterion was adequately met. 

• 4 indicates the criterion was substantially met. 

• 5 indicates the criterion was completely met. 

It can be helpful for a reviewer first to determine whether the criterion was met and then 
determine how well it was met (3 — adequately met, 4 — substantially met, or 5 — completely 
met), or not met (1 — not met or 2 — partially met). After each criterion, there is space for 
reviewer comments (such as specific examples to support ratings, strengths, or concerns). 

Overall rating 

The overall rating subsection is where a reviewer records an overall holistic rating after 
considering the entire set of instructional materials. For example, if a 3 is recorded across 
all the items, a 3 would be expected in the overall rating section. Examples of ratings and 
comments aligned to items from the rubric are provided in table 1. If it is possible for the 
review team to meet and discuss the ratings, it would be most beneficial for the discussion 
to focus on the overall ratings, with the review team going back to the content sections 
to discuss support for ratings. Alternatively, patterns of strengths and weaknesses at the 
item level for component skills within content areas can be noted (for example, one set of 
instructional materials may be strong in foundational skills and weak in reading compre- 
hension skills, whereas the opposite may be true in another set). 

Discussion of results with education leaders 

Once the rubric is completed, the facilitator may share and discuss the results with state, 
district, or school leaders requesting the instructional materials review. 
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Table 1. Examples of ratings and comments in the rubric 

Item from rubric 

Rating 

Example comment 

Grades K-2 

ID. Phonemic awareness instruction follows a 

developmental progression — for example, phoneme 
isolation (first, final, and medial sounds), blending, 
segmentation, and phoneme deletion; use with tiles then 
with letters [FR practice guide, Rec. 2, #1], 

1 (not met) 

Phonemic awareness was noted on the scope 
and sequence; however, phonemic awareness 

activities were not evident within the lessons 

of the instructional materials. 

IE. Graphemes (letters) are gradually integrated into 
phonemic awareness instruction as students become more 
skilled [FR practice guide, Rec. 2, #1]. 

5 (completely met) 

There is a gradual and consistent link between 
phonemic awareness and phonics activities 
throughout the entire set of materials. 

1J. Materials support instruction that teaches students how 
to decode multisyllabic words by looking for pronounceable 
word parts within them (for example, compound words and 
syllables) [FR practice guide, Rec. 3, #3]. 

1 (not met) 

Materials did not include specific instruction on 
how to decode multisyllabic words. 

Grades 3-5 

2E. Specific texts are included in materials for teaching 
various text structures (for example, sequence, comparison, 
contrast, and cause/effect) to support comprehension and 
careful reading of narrative and informational text [Adol 
practice guide, Rec. 2, #1], 

4 (substantially met) 

There are a considerable number of texts to 
support comprehension. There are graphic 
organizers in the back of the teacher editions 
intended to incorporate this item. Flowever, 
there is no label on the graphic organizer to 
indicate the lessons to which they correspond. 
The addition of where the graphic organizers 
could be used would be helpful to the teacher 
for navigation purposes. 

2L. Materials support instruction that teaches students to 
understand and analyze various points of view for narrative 
text (for example, author, narrator, and characters) and 
informational text (for example, what the author wants 
to explain and multiple accounts of the same event) with 
increasing complexity [Adol practice guide, Rec. 3]. 

2 (partially met) 

This is marginally included in the mini 
lessons of the whole group component. 

Different genres and the author's purpose are 
discussed; however, points of view lessons/ 
activities were sparse. 

3A. Materials include extensive practice with short, focused 
research projects that allow students to have multiple 
experiences with the research process throughout the year 
and facilitate development of the ability to conduct research 
independently [EL practice guide, Rec. 3; Writing practice 
guide, Rec. 2a and 2b]. 

2 (partially met) 

There is some evidence of this, mostly found in 
the ancillary materials (science/social studies 
connections and the activities recommended 

for centers). Flowever, those materials are not 
an expected part of daily or even consistent 

lessons. 

Source: Authors' construction. 
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Rubric: Grades K-2 content 


Directions for reviewers using this rubric 


Record your findings based on the extent to which the criteria were met using the 1-5 rating scale. 

1 2 3 4 5 

The criterion was The criterion was The criterion was The criterion was The criterion was 
not met. partially met. adequately met. substantially met. completely met. 


1. Foundational reading skills 


1.1. Materials include guidance to provide small group, differentiated 1 2 3 4 5 

instruction to students struggling with reading development [RTI practice 

guide]. 

Support your rating: 


Print concepts 

1.2. Materials facilitate understanding that sentences can be broken 
into words and compound words can be broken into separate words [FR 
practice guide, Rec. 2, #1], 

1 

2 

3 

4 

5 


Support your rating: 


Phonological awareness 

1.3. Instruction follows a progression to develop phonological awareness 1 

(for example, syllables, rhyming, alliteration, onset, and rime) [FR practice 
guide, Rec. 2, #1], 

2 

3 

4 

5 


Support your rating: 


Phonemic awareness 

1.4. Phonemic awareness instruction follows a developmental 
progression — for example, phoneme isolation (first, final, and medial 
sounds), blending, segmentation, and phoneme deletion; use with tiles 
then with letters [FR practice guide, Rec. 2, #1], 

1 

2 

3 

4 

5 


Support your rating: 


1.5. Graphemes (letters) are gradually integrated into phonemic 1 2 3 4 5 

awareness instruction as students become more skilled [FR practice 
guide, Rec. 2, #1]. 

Support your rating: 


Phonics 






1.6. Materials present letter-sound correspondences in an explicit and 
sequential fashion (for example, potentially confusing letters like b and d 
are not introduced together, and a few short vowels are introduced early) 
[FR practice guide, Rec. 2, #2], 

1 

2 

3 

4 

5 

Support your rating: 






1.7. Materials support instruction that teaches students how to blend 
letters to write/spell and decode words [FR practice guide, Rec. 3, #1]. 

1 

2 

3 

4 

5 

Support your rating: 
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1.8. Materials support instruction that teaches students how to write/ 1 2 3 4 5 

spell and decode common sound-spelling patterns of English in an 
explicit, sequential fashion (for example, CV, CVC words containing 
digraphs, and vowel teams) [FR practice guide, Rec. 3, #2]. 

Support your rating: 


1.9. Materials support instruction that progresses from simple to more 1 2 3 4 5 

complex sound-spelling patterns and word analysis skills (for example, 

CVC words before CVCC, CCVC, and CVe words and single syllable words 
before multisyllabic words) [FR practice guide, Rec. 3, #3]. 

Support your rating: 


1.10. Materials support instruction that teaches students how to decode 1 2 3 4 5 

multisyllabic words by looking for pronounceable word parts within them (for 
example, compound words and syllables) [FR practice guide, Rec. 3, #3], 

Support your rating: 


1.11. Materials provide instruction in simple prefixes and suffixes [FR 1 2 3 4 5 

practice guide, Rec. 3, #3], 

Support your rating: 


1.12. Materials provide sufficient amount of practice in decoding words 1 2 3 4 5 

in isolation and in decodable text to allow practice of phonic skills [FR 
practice guide, Rec. 3, #4], 

Support your rating: 


1.13. Materials include instruction to teach students to read both regular 1 2 3 4 5 

and irregular high frequency words [FR practice guide, Rec. 3, #5], 

Support your rating: 


1.14. Materials introduce students to non-decodable words essential to 1 2 3 4 5 

the meaning of a passage as whole words [FR practice guide, Rec. 3, #6]. 

Support your rating: 


Fluency 

1.15. Materials provide teachers opportunities to model fluent reading and 
to scaffold and give feedback as students read orally in order to foster 
accurate and efficient word identification [FR practice guide, Rec. 4, #1], 

1 

2 

3 

4 

5 


Support your rating: 


1 2. Reading comprehension for literary and informational texts 





2.1. Texts for each grade band align with complexity requirements and 1 

instructional goals [FR practice guide, Rec. 4, #1; RC practice guide, Rec. 4], 

2 

3 

4 

5 

Support your rating: 
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2.2. Materials provide texts that support frequent review of previously 
taught concepts and words, extended practice, and independent 
application of phonics skills [FR practice guide, Rec. 4]. 

1 

2 

3 

4 

5 

Support your rating: 

2.3. Materials include a sufficient number of texts that allow students 

to read a variety of texts daily with and without feedback to support 
accuracy, fluency, and comprehension [RC practice guide, Rec. 2]. 

1 

2 

3 

4 

5 

Support your rating: 

2.4. Materials include texts for students to learn and practice self- 
monitoring as they read and to self-correct reading errors when they 
occur [FR practice guide, Rec. 4, #2; RC practice guide, Rec. 1]. 

1 

2 

3 

4 

5 

Support your rating: 

2.5. Texts are provided for read-alouds that are above the instructional 
level in complexity and guidance. These texts are used to conduct 
multiple read-alouds for the purpose of building vocabulary and reading 
comprehension with text-specific questions [RC practice guide, Rec. 3 
and 4]. 

1 

2 

3 

4 

5 

Support your rating: 

2.6. Materials provide a balance of texts and instructional time for 
literary and informational texts [RC practice guide, Rec. 4]. 

1 

2 

3 

4 

5 

Support your rating: 

2.7. Once students are able to read independently, materials provide 
additional text for regular independent reading that appeals to students' 
interests to develop both knowledge and love of reading [RC practice 
guide, Rec. 5]. 

1 

2 

3 

4 

5 


Support your rating: 


2.8. Texts provide opportunities for students to build knowledge through 1 2 3 4 5 

reading of specific informational and narrative text [RC practice guide, Rec. 3]. 

Support your rating: 


2.9. Materials cultivate students’ abilities to ask and answer questions 1 2 3 4 5 

based on the text [RC practice guide, Rec. 3]. 

Support your rating: 


2.10. Materials use scaffolding and stimulating questions to engage 1 2 3 4 5 

students in high-quality discussions [RC practice guide, Rec. 3]. 

Support your rating: 
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1 3. Writing development and skills Ijj 

3.1. Materials include opportunities to practice writing words introduced 1 

in reading instruction and use them to write in response to what students 
have read. [EL practice guide, Rec. 1]. 

2 

3 

4 

5 

Support your rating: 

3.2. Materials are designed with activities for students to write about what 1 
they read in both literary and informational text (for example, summaries, 
reactions, analysis or interpretation of text, notes, and ask/answer 
questions) [EL practice guide, Rec. 3; Writing practice guide, Rec. 2b]. 

2 

3 

4 

5 


Support your rating: 


3.3. Materials include opportunities and prompts for students to write 1 2 3 4 5 

opinions, information/explanations, or narratives in response to texts 
read [EL practice guide, Rec. 3; Writing practice guide, Rec. 2b]. 

Support your rating: 


4. Speaking and listening development and skills 


4.1. Materials include opportunities to continue to build oral language 1 2 3 4 5 

and listening skills as students develop the ability to read independently 

(for example, exposure to fiction and informational text read aloud, 

discussions to compare/contrast, and analyze and synthesize 

information in response to text read aloud) [FR practice guide, Rec. 1; EL 

practice guide, Rec. 2]. 

Support your rating: 


5. Language development and skills 


5.1. Materials include activities to engage students in conversations that 1 2 3 4 5 

support comprehension of inferential and narrative language and word 
knowledge using narrative and informational text [FR practice guide, Rec. 

1, #1 and 2], 

Support your rating: 


5.2. Materials provide the opportunity to explicitly teach words or 1 2 3 4 5 

grammatical rules that support content that students are reading or 
learning [FR practice guide, Rec. 1, #3], 

Support your rating: 


5.3. Materials provide the opportunity to teach vocabulary by making 1 2 3 4 5 

connections between a new word and other known words, by relating the 

word to their own experiences, by differentiating between correct and 

incorrect uses of the word, and by generating and answering questions 

that include the word [FR practice guide, Rec. 1, #3]. 

Support your rating: 


11 








5.4. Materials include activities for students to acquire and use grade- 1 2 3 4 5 

appropriate general academic and domain-specific words and phrases, 

including those that signal precise actions, emotions, or states of being 

(for example, quizzed, whined, and stammered) and that are basic to a 

particular topic (for example, wildlife, conservation, and endangered when 

discussing animal preservation) [EL practice guide, Rec. 1], 

Support your rating: 


5.5. Materials include instruction for students to determine or clarify the 
meaning of unknown and multiple meaning words and phrases based on 
grade-appropriate reading and content, choosing flexibly from a range of 
strategies [EL practice guide, Rec. 1]. 

1 

2 

3 

4 

5 

Support your rating: 






5.6. Materials include instruction for students to understand word 

relationships and nuances in word meanings (shades of meaning) [EL 
practice guide, Rec. 1], 

1 

2 

3 

4 

5 

Support your rating: 
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Rubric: Grades K-2 overall rating 


Directions for reviewers using this rubric 


Record your findings based on the extent to which the criteria were met using the 1-5 rating scale. 


1 

2 

3 

4 

5 

The criterion was 

The criterion was 

The criterion was 

The criterion was 

The criterion was 

not met. 

partially met. 

adequately met. 

substantially met. 

completely met. 

Title: 


Publisher: 


1. The content is aligned with foundational reading skills for the grade. 1 2 3 4 5 

Support your rating: 


2. The content is aligned with reading comprehension for literary and 1 2 3 4 5 

informational texts for the grade. 

Support your rating: 


3. The content is aligned with writing development and skills for the 1 2 3 4 5 

grade. 

Support your rating: 


4. The content is aligned with speaking and listening development and 1 2 3 4 5 

skills for the grade. 

Support your rating: 


5. The content is aligned with language development and skills for the 1 2 3 4 5 

grade. 

Support your rating: 


Total: 


13 




Rubric: Grades 3-5 content 


Directions for reviewers using this rubric 


Record your findings based on the extent to which the criteria were met using the 1-5 rating scale. 

1 2 3 4 5 

The criterion was The criterion was The criterion was The criterion was The criterion was 
not met. partially met. adequately met. substantially met. completely met. 


1. Foundational reading skills 


1.1. Materials support development of advanced word analysis skills 1 2 3 4 5 

(for example, suffixes, prefixes, Greek and Latin roots, and syllabication 
patterns) [EL practice guide, Rec. 1]. 

Support your rating: 


1.2. Materials include guidance to provide small group, differentiated 1 2 3 4 5 

instruction to students struggling with reading development [RTI practice 
guide, Rees. 2 and 3; EL practice guide, Rec. 4]. 

Support your rating: 


1 2. Reading comprehension for literary and informational texts and text complexity 1 

2.1. The text and text complexity are appropriate for the reading level of 
students [Adol practice guide, Rec. 2]. 

1 2 

3 

4 

5 


Support your rating: 


2.2. Materials provide students extensive opportunities to encounter and 1 2 3 4 5 

comprehend grade-level text [Adol practice guide, Rec. 2]. 

Support your rating: 


2.3. Materials incorporate texts that require a careful and purposeful 
reading and re-reading [Adol practice guide, Rec. 3, #1]. 

1 

2 

3 

4 

5 

Support your rating: 






2.4. Additional materials markedly increase the opportunity for regular 
independent reading of texts that appeal to students' interests to 
develop both knowledge and love of reading [Adol practice guide, Rec. 4]. 

1 

2 

3 

4 

5 

Support your rating: 







2.5. Specific texts are included in materials for teaching various text 1 2 3 4 5 

structures (for example, sequence, comparison, contrast, and cause/ 
effect) to support comprehension and careful reading of narrative and 
informational text [Adol practice guide, Rec. 2, #1). 


Support your rating: 






2.6. Materials contain questions and tasks that require students to use 
text-based evidence (including making inferences) [Adol practice guide, 

Rec. 2 and 3]. 

1 

2 

3 

4 

5 

Support your rating: 
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2.7. Material cultivates student engagement in reading text carefully [Adol 1 2 3 4 5 

practice guide, Rec. 4], 

Support your rating: 


2.8. Texts provide opportunity for students to build knowledge through 1 2 3 4 5 

reading and extended discussion [Adol practice guide, Rec. 3], 

Support your rating: 


2.9. Materials include questions and tasks that require students to 1 2 3 4 5 

analyze information and evidence focused on the meaning of texts [Adol 
practice guide, Rec. 3]. 

Support your rating: 


2.10. Materials support instruction that teaches students to identify 1 2 3 4 5 

and describe or explain ideas for narrative text (for example, main idea 

and theme) and informational text (for example, connections between 

ideas and concepts) in a progressively more complex manner [EL practice 

guide, Rec. 2; Adol practice guide, Rec. 2 and 3]. 

Support your rating: 


2.11. Materials support instruction that teaches students to use the 1 2 3 4 5 

features of text to gain meaning from narrative text (for example, how 

chapters and scenes are used in types of literature) and informational 

text (for example, use of illustrations and graphs, and structural 

elements) with increasing complexity [Adol practice guide, Rec. 5]. 

Support your rating: 


2.12. Materials support instruction that teaches students to understand 1 2 3 4 5 

and analyze various points of view for narrative text (for example, author, 

narrator, and characters) and informational text (for example, what the 

author wants to explain and multiple accounts of the same event) with 

increasing complexity [Adol practice guide, Rec. 3]. 

Support your rating: 


2.13. Materials support instruction that teaches students to compare 1 2 3 4 5 

and contrast or analyze information within and across narrative text 

(for example, compare and contrast two versions of the same story and 

compare and contrast stories in the same genre) and informational text 

(for example, identify similarities and differences between two texts on 

the same topic and integrate information from several texts on the same 

topic to write about the subject) with increasing complexity [Adol practice 

guide, Rec. 2]. 

Support your rating: 


15 


3. Writing development and skills 


3.1. Materials include extensive practice with short, focused research 1 2 3 4 5 

projects that allow students to have multiple experiences with the 
research process throughout the year and facilitate development of the 
ability to conduct research independently [EL practice guide, Rec. 3; 

Writing practice guide, Rec. 2a and 2b]. 

Support your rating: 


3.2. Materials include activities that require students to analyze and 1 2 3 4 5 

synthesize text sources and present the analysis using well-defended 
claims and clear information [EL practice guide, Rec. 3; Writing practice 
guide, Rec. 2b], 

Support your rating: 


3.3. Materials provide instruction in different text structures (for 1 2 3 4 5 

example, sequence, comparison, contrast, and cause/effect) and place 
a focus on argument and informative writing based on texts with these 
structures [EL practice guide, Rec. 3; Writing practice guide, Rec. 2b]. 

Support your rating: 


3.4. Materials include activities that provide opportunities to write 1 2 3 4 5 

routinely over extended time frames (time for research, reflection, and 

revision) and shorter time frames (a single sitting or a day or two) for a 

range of discipline-specific tasks, purposes, and audiences [EL practice 

guide, Rec. 3; Writing practice guide, Rec. 1 and 2b], 

Support your rating: 


1 4. Speaking and listening development and skills j 

4.1. Materials include opportunities to continue to build oral language and 1 
listening skills as students determine main ideas and supporting details, 
paraphrase, and summarize texts read [EL practice guide, Rec. 2]. 

2 

3 

4 

5 


Support your rating: 


4.2. Materials use multimedia and technology to support and engage 1 2 3 4 5 

students in understanding and verbally expressing details and themes in 
a text [EL practice guide, Rec. 2]. 

Support your rating: 


1 5. Language development and skills 





5.1. Materials teach academic vocabulary prevalent in complex texts in 1 

the context of listening and reading activities [EL practice guide, Rec. 1]. 

2 

3 

4 

5 

Support your rating: 
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5.2. Materials include activities for students to acquire and use grade- 1 2 3 4 5 

appropriate general academic and domain-specific words and phrases, 

including those that signal precise actions, emotions, or states of being 

(for example, quizzed, whined, and stammered) and that are basic to a 

particular topic (for example, wildlife, conservation, and endangered when 

discussing animal preservation) [EL practice guide, Rec. 1], 

Support your rating: 


5.3. Materials include instruction for students to determine or clarify the 1 2 3 4 5 

meaning of unknown and multiple meaning words and phrases based on 
grade-appropriate reading and content, choosing flexibly from a range of 
strategies [EL practice guide, Rec. 1]. 


Support your rating: 





5.4. Materials include instruction for students to understand figurative 1 

language, word relationships, and nuances in word meanings [EL practice 
guide, Rec. 1]. 

2 

3 

4 

5 

Support your rating: 
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Rubric: Grades 3-5 overall rating 


Directions for reviewers using this rubric 


Record your findings based on the extent to which the criteria were met using the 1-5 rating scale. 


1 

2 

3 

4 

5 

The criterion was 

The criterion was 

The criterion was 

The criterion was 

The criterion was 

not met. 

partially met. 

adequately met. 

substantially met. 

completely met. 

Title: 


Publisher: 


1. The content is aligned with foundational reading skills for the grade. 1 2 3 4 5 

Support your rating: 


2. The content is aligned with reading comprehension for literary and 1 2 3 4 5 

informational texts and text complexity for the grade. 

Support your rating: 


3. The content is aligned with writing development and skills for the 1 2 3 4 5 

grade. 

Support your rating: 


4. The content is aligned with speaking and listening development and 1 2 3 4 5 

skills for the grade. 

Support your rating: 


5. The content is aligned with language development and skills for the 1 2 3 4 5 

grade. 

Support your rating: 


Total: 


18 




Appendix A. Calculating inter-rater reliability 


Before the review process begins, it is important that each set of instructional materials be 
assigned to two different reviewers in order to determine the degree to which they give con- 
sistent estimates (that is, in order to determine inter-rater reliability). Should the reviewers 
disagree, a third reviewer serves as the tiebreaker. It is recommended that Krippendorff’s 
alpha (Kalpha; Krippendorff, 2004) be used to calculate inter-rater reliability for the rubric. 
Kalpha’s advantages include its ability to measure reliability for judgments made at any 
level of measurement, with any number of reviewers, and with or without missing data. 
Additionally, Kalpha takes into account the possibility that agreement among reviewers 
may occur by chance. Values of Kalpha range from -1.00 to 1.00, where -1.00 indicates 
perfect disagreement below chance (that is, disagreement occurred systematically) and 1 
indicates perfect agreement among reviewers. Krippendorff (2004) suggests that reviewers 
reach a Kalpha greater than or equal to .80 to be considered reliable. 
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The Regional Educational Laboratory Program produces 7 types of reports 



Making Connections 

Studies of correlational relationships 


Making an Impact 

Studies of cause and effect 


What’s Happening 

Descriptions of policies, programs, implementation status, or data trends 

o 

What’s Known 

Summaries of previous research 


Stated Briefly 

Summaries of research findings for specific audiences 

i^k 

Applied Research Methods 

Research methods for educational settings 


Tools 

Help for planning, gathering, analyzing, or reporting data or research 



